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.ABSTRACT ■ # ' , r ' 

This paper 'addresses several issues in quantitative 
research that educational researchers should examine with more care. 
While the purposes of experimentation is td determine causality, the 
study of causal relations is difficult and problematic. Computational 
and conceptual errors in statistical analysis seem limited only by 
the ^creativity of the researcher* The problem of evidence that \ 
contradicts theory is too often solved by throwing out the data. or 
renaming the facts. While researchers have volunteered to improve 
education, thexiniposit ion of a research .finding on all children 
everywhere regardless of the lack of evidence or the presence of 
questionable evidence A is at best a mistake that might not be able to 
be remedied later. (BS) 
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^ My task--and I assure you that it was^-is first <to critique "quarN*, 

titative experimentalism, " a term I don't readily understand, and 
second, to'do so in no more than fifteen minutes. ^Although' I have been 
warned to exceed neither the time nor my knowledge limits - y only the 
first restriction can be' completely controlled. 

However, it is because of time restrictions that I have chosen not 
to discus's the usual criticisms of quantitative methods with which we 
are all so familiar: the lack ^f isomorphism between measurement and 
"reality, 11 whether reality can eve^r be known epistemologically, whether 
any or all educational and psycho K^gical constructs are measured by 
ordinal or interval scales (and whether or not it mak^s any differ- 
ence), and whether we should accept the .05, the .01, or the .001 



significance level. I willw eliminate temptations to discuss both 
? 

determinism and the uniformity of nature as /topics that require more 

time than we have, Instead, I W ^J- try\to address some issues that we, 

/ v " 

as researchers, .should examine vith more care than we have in the past. 

I ^ We are told that the purpose of an experiment is to "determine 

* v. 

"causal" (relations. In fact, I have so stated and in pr.int, but I have 
^ always included causal irf^uote.s or in italics.* I haven't done this 

q because I understand the complexities of that term; rather, I have done 

so because I don t understand it at all. Let me provide an example 

V ' » 
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cited by Robert Morison (1960) some twenty-five years ago, I have re- 
ferred to Morison on oth^r^o|ccasions because he is one ot the few 
people who seems to realize the beauty of quantification when it is 
combined with theory and just how ugly quantification can be when it 
tries t^> pass as disguised scientism. In discussing "cause" and "ef- 
'fect/' Morison makes the point that Jhe Cause of a disease has general - 
ly been thought to be whatever it t is that could--at some given time and 
place-- ameliorate the disease's symptoms. For example, the medifeval 
physicians believed' that malaria was caused by bad\air in lowlands (and 
thus the term mala aria ) . v The lowlands were{ the cause since malarial 
symptoms could l?e reduced or avoided by building 'on hilltops. That 
cause remained undisturbed \titil quinine was introduced into Europe 
from South America'. ^Since quinine could counter th£ symptoms of mal- 
aria no matter where one lived, .quinine must be acting on the body to 
rid it of that disease. By the end of the nineteenth century, the 
malarial ^parasite^ was discovered in the blood of those suffering with 
malareal symptoms, and the parasite became the causal agent. Quinine, 
evidently, helped rid. the body of this parasite'. Later, it was discov- 

T * 

t 

ered that the Anopholes mosquito actually transmitted the disease' and 
was, therefore, its cause. The causal chain extended from location 
Rowlands-), parasite, -and mosquito. 

The story is not quite over. Malarial epidemics rarely occur to- 
day even though little has been done to eradicate the Anopholes mosqui- 
to. The Boston marshes still produce mosquitoes that are capable of 
transmitting the parasite, but no local cases of malaria^frSve occurred. " 
According to Morison, it is now believed ""that epidemic malaria is the 



( . 

result of a nic<|Jy balanced set of ^social and economic, as well as bio- 
logical, factors, each one of which has to be pre^nffc at tlje approprt 1 

ate level 1 ' (page 194). This conclusion might sound more familiar to us 

/ / / * 

if we substituted 3 term such as delinquency for epidemic malaria* 

And since just about everything is "caused" by sdfcial, economic, and 

ft " v * 

bi^lo^ical factors that operate together^in unknown amounts and wavs . 

that Tfeaves "modern" researchers on about the same level of knowledge 

as possessed by their grecft grandparents. Indeed, I once heard research 

characterized as the search for evidence to* prove what your grandmother 

knew all along. 

John Stuart Mill, the 19th century philosopher, proposed five 
methods for studying causality. His method of agreement shows the dif- 
ficulty in studying causal relationships: 

If several instances of ary even|^ have only 
one thing in Common, that thing is the cause 
A of the event. \ \ 

Although this proposition at first seems reasonable, it is not without 

its problems. ~~-€Jfyifeider an experiment in which ninety men had volun^- 

teered to participate in a study on the effects of alcohol. One-t^ilrd 

yere given scotch and water, an equal number were given bourbon and 

water, and the la£t group received vodkd and water. Every man in every 

; \ 

group got tfip-roaring drunk followed by symptoms we al\ knpw only too 
well. The K concIusion: avoid water., when drinking alcohol. I once asked 
,s^idents in an introductory course in jresea^rclj, methods to critique that 
hypothetical study. ^1 must admit th^t I was more than a li/tle ()ur- 
.prised wh en onb student--in all ser iousne6^6f-argued that the study was 



poorly designed because it should have been replicated using school-age 

y 

children. 

Obviously the alcohol study was flawed by having more than "one 
thing in common, 11 in vhich case Milljs canon does not apply. All mejx 
had water, in. addition to alcohol, ajt)d we all know that water does not 



cause inebriation. Or perhaps it does.** Many yearS J ^/as going tc^ 
school and teaching an introductory psychology class ffc adult 

X J 

education. At my request, a dentist friend ordered some nembutal 
placebos for me. I didn't realize that- I would be dispensing drugs 
without a license in which case I had only anticipated a< current trend. 
That evening ^n class, I randomly, assigned half of my volunteers to 
take^tne placebo, and I described vividly how students in other class- 
es had 'fallen asleep "on the floor. No one was permitted to drive home, 
and everyone agreed not tjtf sue me or the school district in which I 
worked. After the <coffee break I returned to the room to find the 
experimental grpup snoring peacefully on the floor. Evidently, even 

I ' 

placebos .have an effect as more recent studies have suggested. \ Whether 
placebos are causal agents or not, we can always resurreo^t the law c}f 
plrsimony which argues that of several equally good hypotheses Science 
will tentatively accent the simplest. .That makes good -sense if we 
could only recognize equally good and simple hypotheses. / 

Perhaps we should describe just one more experiment that can be 
conducted under careful laboratory fcond^tions. ' In this study, ' (he 

experimenter wanted to know if fleas coulcfHbe conditioned. -Fleas, by 

i. * 

i\hp way., have dix legs, and for the purpose of this experiment it was 
necessary to remove their wings*. In classical conditioning the candi- 
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tioned stimulus precedes the unconditioned 'stimulus so the experimenter 
quite properly rang a bell and cut off one leg of* the flea. It jumped.^ 
The bell was rung again, and again the flea jumped* and another leg was 

• , ■ j 

removed. This procedure was repeated four mdre times, and at the end of 
the experiment the conclusion was reached that ringing bells cause 
fleas to become deaf% Since these results can be replicated easily 
and without the need for any high-powered statistics, we have a re- 
liable finding that we cannot blame on faulty statistics. 

Statistics forms an important model in education, and it y is dis- 

J* 

tressingUn the least to observe how poorly statistical analyses can be 

performed. Some ^years ago Quinn McNemar (1960) reported on what he 

called "an astounding^ fallacious significance ^evel": 

a... .psychologist inflated his sample size 36 fold: 
that is, he had 36 observations on each of 45 cases, 
leading to 900 observations whicl) were then treated 
as independent for the chi square analysis. This js 
one way of getting high* statistical significance 
with little prospect 4hat similar results' wil£ be 
found'by those who replicate the study [note: unless, 
- 4 of course,, this becomes standard* practice] . 



*McNemar could have ended the sentence there. 
McNemar was right in being astonished regarding the statistical 
analysis of these data. So many statistical errors'can be found in p\xj)- 
lished studies^ that one can only imagine the number that occu,r on doc-' 
toral dissertations that fortunately never get out of the library. I 
will not bore you with lists of these errors, but they are there and in 
large numbers. "Computational and conceptual errors seem limited only 
by the creativity of the "researcher." Zir part, computers can be 
blamed for some of these problems by enticing students intc working 



mechanically./ One student, after entering only 2-digit numbers for the 

t o 

better . part of a day, reported a mean of^.113.74 without questioning 

these astounding-results. It is easy to disregard any feelings for the 

data or for the effects of experimental procedures when researchers are 

surrounded by mechanical and electronic gadgets that serve little 

purpose except perhaps to help them exchange what is important- for what 

can be obtained with the least -effort. and most money. 
• * s~ 

Students have learned their statistical lessees badly, and they 

carry out their perceived responsibilities too well. If the null hypo- 
thesis cannot be rejected with 30 or 40 persons in each experimental 
and control condition, everyone knows ohat the ."solution" ,is to in- 
crease N until significance is reached. The motto must be' something 
like significan ce no matter, what i This convoluted reasoning Begins 
with the premise that no "two populations are ever identical; therefore, 
there must be a difference between them that should be reflected in the 
^magnitudes of the treatment means. If that reflection happens to be 
missing, 'some ingenuity is needed to force the results to^ome out as 
thfey ate supposed to do. Maier's Law (i960) states that' r facts do 
not conform to the theory, they must be disposed of." I am reminded of 
some types'of test scaling procedures that must have invoked the latent 
spirit of that law. \ 

Like all good "laws," Maier's has corollary attacks that get right 
to the heart and can be invoked should some evidence be allowed to 

! contradict a pet or petty theory. Besides throwing out the data, which 

v 

is one approach to a problem, another good procedure is to rename the 



facts. Maier provides an \example showing that potentially embarrass- 
ing behavior to, learning theorists who insist that reinforcement is 
i 

necessary for learning to occur can be handled quite easily by calling 

the unlearned behavior "imprinting" and not? learning. In this way, 

whatever fails to support some favored position can be retained without 

having % to accept "innate behavior." "Maier also suggests that one good 

way to avoid explanations of events is to given them a title: 

\ For example, a lecturer in describing the habits of people 
living near the North Pole toUl his audience how children ate 
blubber as if it were a^delicacy. Later a questioner asked 
the speaker why these chfydren liked a food that would not be 
attractive to children living here, f The lecturer replied 
ttjat this was so because the children were Eskimos. The 
questioner' replied, "0t\, I see" and was satisfied? In a 
similar manner \ the word "catharsis" explains why we feel 
better after expressing pent-up feelings', (p. 209) 

V 

Another good method for gaining consensus among researchers is to 
C 

express some position mathematical]Ly--as a formula. It may say no more 
or no less than what could be said in understandable English, but the 
very appearance of mathematical symbols will do much to quash con: ro- 
versy . 

Researchers have volunteered to improve education or they have 

been persuaded to do so for the most humane of reasons. Nonetheless, 

it is not the business of researchers to change a world they do not yet 

understand and which may, in not very many years, give them cause for 

concern and possibly regret, fo improve anything or anyone assumes that 

we , know where we want to go, and^ I am not convinced that we have the 

/ 

right to modify behavior (assuming that we can) just because it i^ 
convenient to do so or because we believe Chat we have consensus or 
superior knowledge to fall back on to justify our actions. The purpose 
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of ^research is to obtain reliable knowledge, and we may choose to do 

nothing with that knowledge or we may prefer to act o In either 

case it will not benefit our cause to make sweeping generalizations 

that supposedly apply to all children. The old "new math" was perpe- 

trated on schools and students all over the country before it was_ 

tested at all. At the other extreme we can find statements glorifying 

the deity of ATI (aptitude by treatment interactions). It has been 

eight years since Cronbach and Snow witrned us against believing that we 

now have (or will soon obtain) instructional guidelines from the ATI 

('research. Unfortunately, I can think^of few examples where solid re- 

i 

search evidence has changed the public schools; I can think of numer- 
al 

ous examples where research has been used to defend or to argue \againstf 
the wholesale application of an innovation. Quantitative jresearclT pro- 
vides a meeting ground for differing positions that can be investigated 

empirically regardless of whether or not they provide any amelioriza- 

t » 

tion of some applied problem. Educators can refuse to implement inno- 
vations regardless of their efficacy if those innovations might lead to 
social injustice, excessive costs, or perceived negative effects. What 
should not' be demanded of the quantitative researcher is selected evid- 
ence to support some biased position--a demand that is only thinly dis- 

guised bribery with the payoff being money, recognition, additional 

1 

time, i more space, and new equipment. This misuse of evidence is ser- 
ious because it is so widespread and because it is not recognized as a 
violation by either of fentier-- the one who offers the bribe and t^e one 
who is willing to accept it. The imposition of a research finding on 



8 9 ^ 

i 



all children everywhere regardless of' the. i^ck of evidence or the pres- 
ence of questionable evidence is at best an ethical mistake that might , 
not be 'able to ^e remedied late*r. With our curt^nt state of knowledge, 
we can ask teachers to try new approaches wfien older "solutions", have 

not worked. That |hey might refuse to do so is not only reasonable, but 

t 

it could prevent us from misapplying our own research findings* 
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